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IMPROVEMENTS IN OR RET ATING TO IMAGE RE GISTRATION 

The present invention relates to the registration of images, that is to say the 
process in which two different images are compared to find how they match each 
5 other, and are then displayed superimposed one on the other. 

The registration of different images (also often called fusion of images) is 
useful in a variety of fields. The images being compared and superimposed could be 
images of the same object acquired using different modalities, which thus show up 
different features of interest. The fact that different features of interest are shown by 
1 0 the two modalities is useful in itself, but the usefolness can be enhanced by 

displaying the two images in superimposition. Examples of this technique might be 
the fusion of an infrared image with a visible light image, for instance in. a 
surveillance, mapping or medical situation, or, particularly in the medical field, the 
combination of two different modality images such as magnetic resonance images, 
15 nuclear medicine images, x-ray images, ultrasound images etc. In general this fusion 
of different images assists the interpretation of the images. 

In some situations the two images to be fused are taken at same time, or 
nearly the same time, but in other situations it is useful to fuse images taken at 
different times. For. example, in the medical field it may be useful to fuse an image 
20 taken during one patient examination with an image taken in a different examination, 
for instance six months or a year spaced from the first one. This can assist m 
showing the changes in the patient's condition during that time. The fusion of time- 
separated images arises also in many other fields, such as surveillance and mapping. 
A typical registration (or fusion) technique relies on identifying 
25 corresponding points in the two images and. calculating a transformation which maps 
the pixels of one image to the pixels of another. This may use, for example, the well 
known block matching techniques in which pixels in a block in one image frame are 
compared with pixels in corresponding blocks in a search window in the other image 
frame and the transformation is calculated which minimises a similarity measure in 
30 the intensities in the blocks, such as the sum of square difference. Other techniques 
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based on identification of corresponding shapes in the two images have also been 
proposed. Explanations of different registration techniques are found in, for 
example, US 5,672,877 (AD AC Laboratories), US 5,871,013 (Elscint Limited), and. 
many other text books and published papers. 
5 While such registration techniques are useful, the results can be regarded with 

suspicion by users. This is particularly true where the transformation which maps 
features in one image to features in the other involves not only a rigid movement, but 
also a non-rigid deformation of the image features. Users are typically prepared to 
accept the validity of a rigid movement, such as a translation and/or rotation, between 

1 0 two different images, but the validity of a shape deformation is much less clear. 

Figure 1 of the accompanying drawings illustrates schematically a typical situation. 
An image feature 1 in image frame (a) is found to match an image feature 1' in 
another image frame (b). To map the feature 1' to the original feature 1 it is necessary 
to perform a rigid displacement in the direction of arrow D in image (b), but it can be 

1 5 seen that there is also a shape change required because the right hand side of the 
object is stretched in image (b). Combining the rigid displacement and the 
deformation results in the fused image (c), but it can be seen that there is a resulting 
deformation field in the right hand part of the feature 1" in the fused image (c). 
Particularly in the medical field the concept of deformation like this is regarded with 

20 great scepticism by clinicians as they fear the consequences of an erroneous 

distortion: a stretched or shrunk functional image could lead the clinician to under- or 
over-estimate the extent of a diseased area, and lead to an inappropriate treatment 
with potentially dramatic consequences. 

The present invention provides an image registration, or fusion, method in 

25 which a confidence measure can also be displayed to the user to give the user an idea 
of the quality of registration. This confidence measure is calculated from the 
registration process. The measure of confidence can be, for example, the degree of 
transformation required to perform the mapping, and preferably be based on the 
degree of non-rigid deformation in the transformation. Thus the confidence measure 

30 may exclude rigid motions and represent only the magnitude of the local 
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deformation. The measure can also be based on the local change of volume implied 
by the mapping transformation from one image to the other. 

Preferably the measure is selectively displayed in response to user input and it 
may be displayed as a visually distinguishable overlay on the display of the fused 
5 images. It may comprise a colour overlay with the colour or intensity (or both) 
indicating the measure of confidence, or the same could be achieved with a 
monochrome overlay whose grey level represents the measure. Alternatively, a 
symbol, such as a circle, can be displayed at any selected point in the fused image, 
whose size and/or shape, for instance the diameter of the circle, is measure of the 
10 confidence in the registration. Preferably, to avoid cluttering the display, the symbol, 
is only displayed at a -single point selected by the user, for example in response to the 
user "clicking" at the selected point on the screen. 

The method is particularly applicable to fused medical images, though it is 
also applicable in other fields where images are registered, such as surveillance, 
15 mapping etc. 

The invention may conveniently be embodied as a computer program 
comprising program code means for executing the method, and the invention extends 
• to a storage or transmission medium encoding the program and to an image 
processing and display apparatus which performs the method. 
20 * • The invention will be further described by way of example with referee 

the accompanying drawings, in which:- 

Figure 1 is a schematic diagram of an image registration or fusion process; 
Figure 2 is a flow diagram of an embodiment of the invention; 
Figure 3 illustrates an original MR image of the brain; 
25 Figure 4 illustrates an original PET image of the brain; 

Figure 5 illustrates the result of applying a non-rigid transformation to the 
PET image of Figure 4 so that it will register with the MR image of Figure 3 ; 

Figure 6 illustrates the original MR image superimposed with the deformed 
PET image in registration with it; 
30 . Figure 7 illustrates the result of applying only a rigid transformation to the 
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PET image of Figure 4 so that it will register with the MR image of Figure 3; 

Figure 8 illustrates a display of a fused image in accordance with one 
embodiment of the invention; and 

Figure 9 illustrates a display of a fused image in accordance with another 
5 embodiment of the invention. 

As indicated in Figure 2 a typical image fusion or registration process 
involves at step 21 the input of two images. These may be individual image frames 
in static imaging, or could be image sequences in a dynamic imaging application. 
The two images are compared in step 22 and the transformation which best maps 
10 features from one image onto corresponding features of the other is calculated. This 
transformation is, in essence, a mapping of pixels in one image to pixels in the other. 
Taking, as an example, a three dimensional image this may be expressed as 

(x 1 ^ 1 ,z 1 )=RIG(^ 2 ^ 2 ,z 2 )+DEF(x 25 ^,z 2 ) (i) 

15 

where F represents the mapping transformation. Typically this transformation 
may include a rigid movement and a deformation viz:- 

{x l3 y l9 zj= J?(x 2 ,y 2 ,z 2 ) . (2) 

.20 

The rigid part of the movement may be a translation and a rotation, namely:- 

(xjyj , z x ) = TRANS(x 2 , y 2 , z 2 ) + ROT(* 2 , y 2 , * 2 ) + DEF(* 2 , y 2 , z 2 ) (3) 

Figures 3, 4, 5 and 6 illustrate an example of this method applied to brain 
25 imaging. Figures 3 and 4 illustrate respectively the original MR and PET images of 
the brain. Figure 5 illustrates the result of applying a non-rigid transformation to the 
PET image of Figure 4 so that it will register with the MR image of Figure 3 and 
Figure 6 illustrates the original MR image superimposed with the deformed PET 
image in registration with it. The advantages of being able to see at a glance the 
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information from both imaging modalities are clear. By way of a comparison, Figure 
7 illustrates the result of superimposing a version of the PET image of Figure 4 onto 
the MR image of Figure 3 with only a rigid deformation. 

In accordance with this embodiment of the invention the size of the 
deformable part of the transformation DEF is regarded as a measure of the 
disagreement between the rigid registration (RIG) and the deformable registration 
(RIG+DEF). Thus a confidence measure Mis calculated from the deformable part of 
the transformation. As one example the confidence measure Mmay be simply the 
magnitude (norm) of the local displacement. That is to say :- 



10 



M= |dEF(x 2 ,^ 2 ,z 2 )| 



(4) 



Alternatively, the confidence measure may be calculated as the determinant of 
the local Jacobian of the transformation, which defines the local stretching (change of 
15 volume) at a particular location. Soif:- 

*i = F x (x 2 ,y 2 ,z 2 ) 

yi = F y (x 2 ,y 2 ,z 2 ) ( 5 ) 

*i = F 2 (x 2 ,y 2 ,z 2 ) 



Then the measure Mbecomes:- 
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(6) 



It is also possible to base the measure on the value of the similarity function 




-6- 

such as cross-correlation, mutual information, correlation ratio or the like (see for 
example A. Roche, X. Pennec, G. Malandain, and N. Ayache. Rigid Registration of 
3D Ultrasound with MR Images: a New Approach Combining Intensity and Gradient 
Information. IEEE Transactions on Medical Imaging, 20(10):1038--1049, 
5 October 2001) used in matching local blocks in the images, or to combine these 
various measure together to form a normalised estimate of the "confidence" in the 
registration process. 

Once the measure has been calculated it can be displayed over the fused 
image. One way of displaying it is, in response to the user "clicking" at a certain 
10 point on the display, to display a circle whose diameter represents the value of the 
confidence measure. Figure 8 illustrates this applied to the fused image of Figure 6. 
The user can "point" to different positions on the image using the perpendicularly- 
intersecting cross-hairs (typically controlled by a pointing device such as a computer 
mouse), and "clicking" at the selected position causes a circle to be displayed as 
1 5 shown. The larger the circle the more non-rigid deformation has occurred and so the 
less agreement there is between the rigid and non-rigid registration. Thus the more 
careful the clinician should be while reviewing this fusion result. On the other hand, 
in areas where the circle has a small diameter, the registration has been basically a 
rigid movement, and thus the result of the fusion can be regarded as more reliable. 
20 Figure 9 shows an alternative way of displaying the confidence measure on the 

fused image of Figure 6. In Figure 9 the overlay is a transparent colour wash (though 
shown in black and white in Figure 9) whose colour and intensity are directly related 
to the value of the confidence measure. For example, green of low intensity may be 
used where the confidence is high (i.e. the amount of non-rigid deformation is low) 
25 whereas red, growing more intense, can be used as the amount of non-rigid 

deformation increases. It can be seen that the confidence decreases towards the left of 
the image where a high non-rigid deformation was required to register the two 
images. 



CLAIMS 



1 . A method of displaying two images in registration with each other comprising the 
5 steps of comparing the two images to each other, calculating a transformation which 
maps features in one image to corresponding features in the other, displaying the two 
images in superimposition based on the transformation, and displaying overlaid on th 
two images a measure of the confidence in the registration. 

10 2. A method according to claim 1 wherein said measure of confidence is calculated 
from the degree of transformation required to perform said mapping. 

3. A method according to claim 2 wherein said measure of confidence is calculated 
from the degree of non-rigid deformation in said calculated transformation. 

15 

4. A method according to claim 2 or 3 wherein said measure is calculated excluding 
rigid motions. 

5. A method according to claim 2, 3 or 4 wherein said measure of confidence is 
20 calculated from the magnitude of the local deformation in said transformation. 

6. A method according to claim 2, 3, 4 or 5 wherein said measure of confidence is. 
calculated from the local change of volume implied by the transformation. 

25 7. A method according to any one of the preceding claims wherein the measure is 
selectively displayed in response to user input. 

8. A method according to any one of the preceding claims wherein the measure is 
displayed as a visually distinguishable overlay on the two images, the visual 
3 0 properties of the overlay at any point being based on the said measure. 
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9. A method according to claim 8 wherein the colour of the visually distinguishable 
overlay is varied in dependence on said measure. 

10. A method according to claim 8 or 9 wherein the intensity of the visually 
5 distinguishable overlay is varied in dependence on said measure. 

1 1. A method according to claim 8 wherein the grey-level of the visually 
distinguishable overlay is varied in dependence on said measure. 

10 12. A method according to claim 8 wherein the visually distinguishable overlay . 
comprises a symbol whose size and/or shape at a selected display point depends on 
said measure. 

13. A method according to claim 12 wherein the symbol comprises a circle whose 
1 5 diameter at a selected display point depends on said measure. 

14. A method according to claim 12 or 13 wherein the symbol is displayed at any time 
only at a single selected display point. 

20 1 5 . A method according to any one of the preceding claims wherein the images are 
medical images. 

16.. A computer program comprising program code means for executing on a 
programmed computer the method of any one of the preceding claims. 

25 

. 17. A computer-readable storage medium encoding a computer program in 
accordance with claim 16. 

18. An image display apparatus comprising a display, and an image processor 
30 adapted to perform the method of any one of claims 1 to 15. 




-9- 

19. A method of displaying two images in registration with each other, the method 
being substantially as hereinbefore described with reference to and as illustrated in the 
accompanying drawings. 
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ABSTRACT 

IMPROVEMENTS IN OR RELATING TO IMAGE REGISTRATION 

A method of displaying two images in registration with each other in which a 
5 visually distinguishable overlay is also displayed to represent the degree of 

"confidence 11 in the registration process. The degree of confidence may be calculated 
on the basis of the degree of non-rigid deformation needed to register the two images. 
The visually distinguishable overlay can be in the form of a transparent colour wash 
whose colour and/or intensity indicate the level of confidence, or a symbol, e.g. a 
10 circle, whose size represents the degree of confidence. 
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